培训低级的深层神经网络，即使用分解层，特别是社区感兴趣的：它在记忆消耗和训练时间方面提供了对未分离培训的效率。先前的工作集中在预训练的网络的低级近似值和低级空间中的培训中，并提供了其他目标，为所选实践提供了各种临时解释。我们分析了在实践中运作良好的技术，并通过对诸如GPT2之类的模型进行广泛的消融，我们提供了证据表明该领域的共同信念，这暗示着令人兴奋的研究机会仍然需要回答。

对网络规模数据进行培训可能需要几个月的时间。但是，在已经学习或不可学习的冗余和嘈杂点上浪费了很多计算和时间。为了加速训练，我们引入了可减少的持有损失选择（Rho-loss），这是一种简单但原则上的技术，它大致选择了这些训练点，最大程度地减少了模型的概括损失。结果，Rho-loss减轻了现有数据选择方法的弱点：优化文献中的技术通常选择“硬损失”（例如，高损失），但是这种点通常是嘈杂的（不可学习）或更少的任务与任务相关。相反，课程学习优先考虑“简单”的积分，但是一旦学习，就不必对这些要点进行培训。相比之下，Rho-Loss选择了可以学习的点，值得学习的，尚未学习。与先前的艺术相比，Rho-loss火车的步骤要少得多，可以提高准确性，并加快对广泛的数据集，超参数和体系结构（MLP，CNNS和BERT）的培训。在大型Web绑带图像数据集服装1M上，与统一的数据改组相比，步骤少18倍，最终精度的速度少2％。

我们介绍了Goldilocks Selection，这是一种用于更快的模型训练的技术，该技术选择了一系列“恰到好处”的训练点。我们提出了一个信息理论采集函数 - 可还原验证损失 - 并使用小的代理模型-GoldiProx进行计算，以有效地选择培训点，以最大程度地提高有关验证集的信息。我们表明，通常在优化文献中选择的“硬”（例如高损失）点通常是嘈杂的，而“简单”（例如低噪声）样本通常优先考虑课程学习提供更少的信息。此外，具有不确定标签的点（通常是由主动学习的目标）往往与任务相关。相比之下，Goldilocks选择选择了“恰到好处”的点，并且从经验上优于上述方法。此外，选定的序列可以转移到其他体系结构。从业者可以共享并重复使用它，而无需重新创建它。

近年来，最先进神经网络的参数的数量急剧增加。这种对大规模神经网络感兴趣的激增具有促使新的分布式培训策略的发展，从而实现了这种模型。一种这样的策略是模型平行分布式培训。不幸的是，模型 - 并行性遭受资源利用率差，导致资源浪费。在这项工作中，我们改进了最近的理想化模型 - 并行优化设置：本地学习。由于资源利用率差，我们在当地和全球学习之间介绍了一类中介战略，称为联锁反向化。这些策略保留了本地优化的许多计算效率优势，同时恢复全球优化实现的大部分任务性能。我们评估了我们对图像分类的策略和变压器语言模型，发现我们的策略一致地在任务绩效方面出现本地学习，并在培训效率方面进行全球学习。

While the brain connectivity network can inform the understanding and diagnosis of developmental dyslexia, its cause-effect relationships have not yet enough been examined. Employing electroencephalography signals and band-limited white noise stimulus at 4.8 Hz (prosodic-syllabic frequency), we measure the phase Granger causalities among channels to identify differences between dyslexic learners and controls, thereby proposing a method to calculate directional connectivity. As causal relationships run in both directions, we explore three scenarios, namely channels' activity as sources, as sinks, and in total. Our proposed method can be used for both classification and exploratory analysis. In all scenarios, we find confirmation of the established right-lateralized Theta sampling network anomaly, in line with the temporal sampling framework's assumption of oscillatory differences in the Theta and Gamma bands. Further, we show that this anomaly primarily occurs in the causal relationships of channels acting as sinks, where it is significantly more pronounced than when only total activity is observed. In the sink scenario, our classifier obtains 0.84 and 0.88 accuracy and 0.87 and 0.93 AUC for the Theta and Gamma bands, respectively.

Differentiable Architecture Search (DARTS) has attracted considerable attention as a gradient-based Neural Architecture Search (NAS) method. Since the introduction of DARTS, there has been little work done on adapting the action space based on state-of-art architecture design principles for CNNs. In this work, we aim to address this gap by incrementally augmenting the DARTS search space with micro-design changes inspired by ConvNeXt and studying the trade-off between accuracy, evaluation layer count, and computational cost. To this end, we introduce the Pseudo-Inverted Bottleneck conv block intending to reduce the computational footprint of the inverted bottleneck block proposed in ConvNeXt. Our proposed architecture is much less sensitive to evaluation layer count and outperforms a DARTS network with similar size significantly, at layer counts as small as 2. Furthermore, with less layers, not only does it achieve higher accuracy with lower GMACs and parameter count, GradCAM comparisons show that our network is able to better detect distinctive features of target objects compared to DARTS.

We propose an ensemble approach to predict the labels in linear programming word problems. The entity identification and the meaning representation are two types of tasks to be solved in the NL4Opt competition. We propose the ensembleCRF method to identify the named entities for the first task. We found that single models didn't improve for the given task in our analysis. A set of prediction models predict the entities. The generated results are combined to form a consensus result in the ensembleCRF method. We present an ensemble text generator to produce the representation sentences for the second task. We thought of dividing the problem into multiple small tasks due to the overflow in the output. A single model generates different representations based on the prompt. All the generated text is combined to form an ensemble and produce a mathematical meaning of a linear programming problem.

Diabetic Retinopathy (DR) is a leading cause of vision loss in the world, and early DR detection is necessary to prevent vision loss and support an appropriate treatment. In this work, we leverage interactive machine learning and introduce a joint learning framework, termed DRG-Net, to effectively learn both disease grading and multi-lesion segmentation. Our DRG-Net consists of two modules: (i) DRG-AI-System to classify DR Grading, localize lesion areas, and provide visual explanations; (ii) DRG-Expert-Interaction to receive feedback from user-expert and improve the DRG-AI-System. To deal with sparse data, we utilize transfer learning mechanisms to extract invariant feature representations by using Wasserstein distance and adversarial learning-based entropy minimization. Besides, we propose a novel attention strategy at both low- and high-level features to automatically select the most significant lesion information and provide explainable properties. In terms of human interaction, we further develop DRG-Net as a tool that enables expert users to correct the system's predictions, which may then be used to update the system as a whole. Moreover, thanks to the attention mechanism and loss functions constraint between lesion features and classification features, our approach can be robust given a certain level of noise in the feedback of users. We have benchmarked DRG-Net on the two largest DR datasets, i.e., IDRID and FGADR, and compared it to various state-of-the-art deep learning networks. In addition to outperforming other SOTA approaches, DRG-Net is effectively updated using user feedback, even in a weakly-supervised manner.

We present a Machine Learning (ML) study case to illustrate the challenges of clinical translation for a real-time AI-empowered echocardiography system with data of ICU patients in LMICs. Such ML case study includes data preparation, curation and labelling from 2D Ultrasound videos of 31 ICU patients in LMICs and model selection, validation and deployment of three thinner neural networks to classify apical four-chamber view. Results of the ML heuristics showed the promising implementation, validation and application of thinner networks to classify 4CV with limited datasets. We conclude this work mentioning the need for (a) datasets to improve diversity of demographics, diseases, and (b) the need of further investigations of thinner models to be run and implemented in low-cost hardware to be clinically translated in the ICU in LMICs. The code and other resources to reproduce this work are available at https://github.com/vital-ultrasound/ai-assisted-echocardiography-for-low-resource-countries.

This paper deals with the problem of statistical and system heterogeneity in a cross-silo Federated Learning (FL) framework where there exist a limited number of Consumer Internet of Things (CIoT) devices in a smart building. We propose a novel Graph Signal Processing (GSP)-inspired aggregation rule based on graph filtering dubbed ``G-Fedfilt''. The proposed aggregator enables a structured flow of information based on the graph's topology. This behavior allows capturing the interconnection of CIoT devices and training domain-specific models. The embedded graph filter is equipped with a tunable parameter which enables a continuous trade-off between domain-agnostic and domain-specific FL. In the case of domain-agnostic, it forces G-Fedfilt to act similar to the conventional Federated Averaging (FedAvg) aggregation rule. The proposed G-Fedfilt also enables an intrinsic smooth clustering based on the graph connectivity without explicitly specified which further boosts the personalization of the models in the framework. In addition, the proposed scheme enjoys a communication-efficient time-scheduling to alleviate the system heterogeneity. This is accomplished by adaptively adjusting the amount of training data samples and sparsity of the models' gradients to reduce communication desynchronization and latency. Simulation results show that the proposed G-Fedfilt achieves up to $3.99\% $ better classification accuracy than the conventional FedAvg when concerning model personalization on the statistically heterogeneous local datasets, while it is capable of yielding up to $2.41\%$ higher accuracy than FedAvg in the case of testing the generalization of the models.